資料分析 從R 開始

David Chiu (largitdata.com)
david@largitdata.com
2014/10/03

關於我

alt text

  • 大數軟體(largitdata.com)有限公司創辦人

  • 前趨勢科技工程師

  • ywchiu.com

什麼是R

  • AT&T貝爾實驗室暨S語言所發展出來的GNU 專案
  • 提供統計分析與圖形視覺化功能的開源程式語言
  • 使用C, Fortran 編程的函式語言

alt text

為什麼使用R

alt text

R語言是

  • R 是函式語言 (Functional Programming)

    • 給予數學定義,便可以數學函式求得解答
  • R 是直譯式語言 (Interpreted Language)

    • 一行行執行,可直接看到執行結果
  • R 是物件導向語言 (Object Oriented Language)

    • R可以用S3, S4定義物件

最廣泛被用來做資料分析的語言

最受歡迎的語言持續為 R, Python (39%), 及 SQL (37%). SAS 大約在 20%上下.

By Gregory Piatetsky, Aug 27, 2013.

alt text

Kaggle

R 是最廣為Kaggle比賽參與者所使用的語言

http://www.kaggle.com/

alt text

Google 的資料科學家使用 R

alt text

What is your programming language of choice, R, Python or something else?

“I use R, and occasionally matlab, for data analysis. There is a large, active and extremely knowledgeable R community at Google.”

蘋果的資料科學家也使用 R

Expert knowledge of SAS (With Enterprise Guide/Miner) required and candidates with strong knowledge of R will be preferred

alt text

商業版本的 R

  • 在 2007, Revolution Analytics 提供商業版本的R

  • Big Data Appliance, 整合R, Apache Hadoop, Oracle Enterprise Linux, 和 NoSQL 資料庫於 Exadata

Distributed R

alt text

Revolution R

alt text

整合式開發環境 IDE

Rstudio

alt text

網頁應用程式開發

Shiny 讓使用者可以透過互動式網頁介面,在上面直接進行分析

alt text

RPubs

套件管理

CRAN (Comprehensive R Archive Network)

alt text

資料分析

資料分析如同做菜

alt text

R 語言基礎

  • 資料型態 (Vector,List,Factor,Array,Matrix,Data Frame)
  • 自訂函式
  • 控制流程
  • 資料IO
  • 物件導向

內建資料集

使用內建資料

data(iris)
str(iris)
'data.frame':   150 obs. of  5 variables:
 $ Sepal.Length: num  5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
 $ Sepal.Width : num  3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
 $ Petal.Length: num  1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
 $ Petal.Width : num  0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
 $ Species     : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...

資料篩選 (一)

取前五筆包含length 及 width 的資料

Five.Sepal.iris = iris[1:5, c("Sepal.Length", "Sepal.Width")]
Five.Sepal.iris
  Sepal.Length Sepal.Width
1          5.1         3.5
2          4.9         3.0
3          4.7         3.2
4          4.6         3.1
5          5.0         3.6

資料篩選 (二)

可以用條件做篩選

setosa.data = iris[iris$Species=="setosa",1:5]
setosa.data
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1           5.1         3.5          1.4         0.2  setosa
2           4.9         3.0          1.4         0.2  setosa
3           4.7         3.2          1.3         0.2  setosa
4           4.6         3.1          1.5         0.2  setosa
5           5.0         3.6          1.4         0.2  setosa
6           5.4         3.9          1.7         0.4  setosa
7           4.6         3.4          1.4         0.3  setosa
8           5.0         3.4          1.5         0.2  setosa
9           4.4         2.9          1.4         0.2  setosa
10          4.9         3.1          1.5         0.1  setosa
11          5.4         3.7          1.5         0.2  setosa
12          4.8         3.4          1.6         0.2  setosa
13          4.8         3.0          1.4         0.1  setosa
14          4.3         3.0          1.1         0.1  setosa
15          5.8         4.0          1.2         0.2  setosa
16          5.7         4.4          1.5         0.4  setosa
17          5.4         3.9          1.3         0.4  setosa
18          5.1         3.5          1.4         0.3  setosa
19          5.7         3.8          1.7         0.3  setosa
20          5.1         3.8          1.5         0.3  setosa
21          5.4         3.4          1.7         0.2  setosa
22          5.1         3.7          1.5         0.4  setosa
23          4.6         3.6          1.0         0.2  setosa
24          5.1         3.3          1.7         0.5  setosa
25          4.8         3.4          1.9         0.2  setosa
26          5.0         3.0          1.6         0.2  setosa
27          5.0         3.4          1.6         0.4  setosa
28          5.2         3.5          1.5         0.2  setosa
29          5.2         3.4          1.4         0.2  setosa
30          4.7         3.2          1.6         0.2  setosa
31          4.8         3.1          1.6         0.2  setosa
32          5.4         3.4          1.5         0.4  setosa
33          5.2         4.1          1.5         0.1  setosa
34          5.5         4.2          1.4         0.2  setosa
35          4.9         3.1          1.5         0.2  setosa
36          5.0         3.2          1.2         0.2  setosa
37          5.5         3.5          1.3         0.2  setosa
38          4.9         3.6          1.4         0.1  setosa
39          4.4         3.0          1.3         0.2  setosa
40          5.1         3.4          1.5         0.2  setosa
41          5.0         3.5          1.3         0.3  setosa
42          4.5         2.3          1.3         0.3  setosa
43          4.4         3.2          1.3         0.2  setosa
44          5.0         3.5          1.6         0.6  setosa
45          5.1         3.8          1.9         0.4  setosa
46          4.8         3.0          1.4         0.3  setosa
47          5.1         3.8          1.6         0.2  setosa
48          4.6         3.2          1.4         0.2  setosa
49          5.3         3.7          1.5         0.2  setosa
50          5.0         3.3          1.4         0.2  setosa

資料篩選 (三)

使用which 做資料篩選

which(iris$Species=="setosa")
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
[24] 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
[47] 47 48 49 50

使用subset 做資料篩選 (一)

使用subset 取欄位

sepal.data = subset(iris, select=c("Sepal.Length", "Sepal.Width"))
sepal.data
    Sepal.Length Sepal.Width
1            5.1         3.5
2            4.9         3.0
3            4.7         3.2
4            4.6         3.1
5            5.0         3.6
6            5.4         3.9
7            4.6         3.4
8            5.0         3.4
9            4.4         2.9
10           4.9         3.1
11           5.4         3.7
12           4.8         3.4
13           4.8         3.0
14           4.3         3.0
15           5.8         4.0
16           5.7         4.4
17           5.4         3.9
18           5.1         3.5
19           5.7         3.8
20           5.1         3.8
21           5.4         3.4
22           5.1         3.7
23           4.6         3.6
24           5.1         3.3
25           4.8         3.4
26           5.0         3.0
27           5.0         3.4
28           5.2         3.5
29           5.2         3.4
30           4.7         3.2
31           4.8         3.1
32           5.4         3.4
33           5.2         4.1
34           5.5         4.2
35           4.9         3.1
36           5.0         3.2
37           5.5         3.5
38           4.9         3.6
39           4.4         3.0
40           5.1         3.4
41           5.0         3.5
42           4.5         2.3
43           4.4         3.2
44           5.0         3.5
45           5.1         3.8
46           4.8         3.0
47           5.1         3.8
48           4.6         3.2
49           5.3         3.7
50           5.0         3.3
51           7.0         3.2
52           6.4         3.2
53           6.9         3.1
54           5.5         2.3
55           6.5         2.8
56           5.7         2.8
57           6.3         3.3
58           4.9         2.4
59           6.6         2.9
60           5.2         2.7
61           5.0         2.0
62           5.9         3.0
63           6.0         2.2
64           6.1         2.9
65           5.6         2.9
66           6.7         3.1
67           5.6         3.0
68           5.8         2.7
69           6.2         2.2
70           5.6         2.5
71           5.9         3.2
72           6.1         2.8
73           6.3         2.5
74           6.1         2.8
75           6.4         2.9
76           6.6         3.0
77           6.8         2.8
78           6.7         3.0
79           6.0         2.9
80           5.7         2.6
81           5.5         2.4
82           5.5         2.4
83           5.8         2.7
84           6.0         2.7
85           5.4         3.0
86           6.0         3.4
87           6.7         3.1
88           6.3         2.3
89           5.6         3.0
90           5.5         2.5
91           5.5         2.6
92           6.1         3.0
93           5.8         2.6
94           5.0         2.3
95           5.6         2.7
96           5.7         3.0
97           5.7         2.9
98           6.2         2.9
99           5.1         2.5
100          5.7         2.8
101          6.3         3.3
102          5.8         2.7
103          7.1         3.0
104          6.3         2.9
105          6.5         3.0
106          7.6         3.0
107          4.9         2.5
108          7.3         2.9
109          6.7         2.5
110          7.2         3.6
111          6.5         3.2
112          6.4         2.7
113          6.8         3.0
114          5.7         2.5
115          5.8         2.8
116          6.4         3.2
117          6.5         3.0
118          7.7         3.8
119          7.7         2.6
120          6.0         2.2
121          6.9         3.2
122          5.6         2.8
123          7.7         2.8
124          6.3         2.7
125          6.7         3.3
126          7.2         3.2
127          6.2         2.8
128          6.1         3.0
129          6.4         2.8
130          7.2         3.0
131          7.4         2.8
132          7.9         3.8
133          6.4         2.8
134          6.3         2.8
135          6.1         2.6
136          7.7         3.0
137          6.3         3.4
138          6.4         3.1
139          6.0         3.0
140          6.9         3.1
141          6.7         3.1
142          6.9         3.1
143          5.8         2.7
144          6.8         3.2
145          6.7         3.3
146          6.7         3.0
147          6.3         2.5
148          6.5         3.0
149          6.2         3.4
150          5.9         3.0

使用subset 做資料篩選 (二)

以Species 做篩選條件

setosa.data = subset(iris, Species =="setosa")
setosa.data
   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1           5.1         3.5          1.4         0.2  setosa
2           4.9         3.0          1.4         0.2  setosa
3           4.7         3.2          1.3         0.2  setosa
4           4.6         3.1          1.5         0.2  setosa
5           5.0         3.6          1.4         0.2  setosa
6           5.4         3.9          1.7         0.4  setosa
7           4.6         3.4          1.4         0.3  setosa
8           5.0         3.4          1.5         0.2  setosa
9           4.4         2.9          1.4         0.2  setosa
10          4.9         3.1          1.5         0.1  setosa
11          5.4         3.7          1.5         0.2  setosa
12          4.8         3.4          1.6         0.2  setosa
13          4.8         3.0          1.4         0.1  setosa
14          4.3         3.0          1.1         0.1  setosa
15          5.8         4.0          1.2         0.2  setosa
16          5.7         4.4          1.5         0.4  setosa
17          5.4         3.9          1.3         0.4  setosa
18          5.1         3.5          1.4         0.3  setosa
19          5.7         3.8          1.7         0.3  setosa
20          5.1         3.8          1.5         0.3  setosa
21          5.4         3.4          1.7         0.2  setosa
22          5.1         3.7          1.5         0.4  setosa
23          4.6         3.6          1.0         0.2  setosa
24          5.1         3.3          1.7         0.5  setosa
25          4.8         3.4          1.9         0.2  setosa
26          5.0         3.0          1.6         0.2  setosa
27          5.0         3.4          1.6         0.4  setosa
28          5.2         3.5          1.5         0.2  setosa
29          5.2         3.4          1.4         0.2  setosa
30          4.7         3.2          1.6         0.2  setosa
31          4.8         3.1          1.6         0.2  setosa
32          5.4         3.4          1.5         0.4  setosa
33          5.2         4.1          1.5         0.1  setosa
34          5.5         4.2          1.4         0.2  setosa
35          4.9         3.1          1.5         0.2  setosa
36          5.0         3.2          1.2         0.2  setosa
37          5.5         3.5          1.3         0.2  setosa
38          4.9         3.6          1.4         0.1  setosa
39          4.4         3.0          1.3         0.2  setosa
40          5.1         3.4          1.5         0.2  setosa
41          5.0         3.5          1.3         0.3  setosa
42          4.5         2.3          1.3         0.3  setosa
43          4.4         3.2          1.3         0.2  setosa
44          5.0         3.5          1.6         0.6  setosa
45          5.1         3.8          1.9         0.4  setosa
46          4.8         3.0          1.4         0.3  setosa
47          5.1         3.8          1.6         0.2  setosa
48          4.6         3.2          1.4         0.2  setosa
49          5.3         3.7          1.5         0.2  setosa
50          5.0         3.3          1.4         0.2  setosa

使用subset 做資料篩選 (三)

可以做篩選條件的組合

example.data= subset(iris, Petal.Length <=1.4 & Petal.Width >= 0.2, select=Species )
example.data
   Species
1   setosa
2   setosa
3   setosa
5   setosa
7   setosa
9   setosa
15  setosa
17  setosa
18  setosa
23  setosa
29  setosa
34  setosa
36  setosa
37  setosa
39  setosa
41  setosa
42  setosa
43  setosa
46  setosa
48  setosa
50  setosa

資料合併

以Merge 做資料合併

flower.type = data.frame(Species = "setosa", Flower = "iris")
merge(flower.type, iris[1:3,], by ="Species")
  Species Flower Sepal.Length Sepal.Width Petal.Length Petal.Width
1  setosa   iris          5.1         3.5          1.4         0.2
2  setosa   iris          4.9         3.0          1.4         0.2
3  setosa   iris          4.7         3.2          1.3         0.2

資料排序

用order做資料排序

iris[order(iris$Sepal.Length, decreasing = TRUE),]
    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
132          7.9         3.8          6.4         2.0  virginica
118          7.7         3.8          6.7         2.2  virginica
119          7.7         2.6          6.9         2.3  virginica
123          7.7         2.8          6.7         2.0  virginica
136          7.7         3.0          6.1         2.3  virginica
106          7.6         3.0          6.6         2.1  virginica
131          7.4         2.8          6.1         1.9  virginica
108          7.3         2.9          6.3         1.8  virginica
110          7.2         3.6          6.1         2.5  virginica
126          7.2         3.2          6.0         1.8  virginica
130          7.2         3.0          5.8         1.6  virginica
103          7.1         3.0          5.9         2.1  virginica
51           7.0         3.2          4.7         1.4 versicolor
53           6.9         3.1          4.9         1.5 versicolor
121          6.9         3.2          5.7         2.3  virginica
140          6.9         3.1          5.4         2.1  virginica
142          6.9         3.1          5.1         2.3  virginica
77           6.8         2.8          4.8         1.4 versicolor
113          6.8         3.0          5.5         2.1  virginica
144          6.8         3.2          5.9         2.3  virginica
66           6.7         3.1          4.4         1.4 versicolor
78           6.7         3.0          5.0         1.7 versicolor
87           6.7         3.1          4.7         1.5 versicolor
109          6.7         2.5          5.8         1.8  virginica
125          6.7         3.3          5.7         2.1  virginica
141          6.7         3.1          5.6         2.4  virginica
145          6.7         3.3          5.7         2.5  virginica
146          6.7         3.0          5.2         2.3  virginica
59           6.6         2.9          4.6         1.3 versicolor
76           6.6         3.0          4.4         1.4 versicolor
55           6.5         2.8          4.6         1.5 versicolor
105          6.5         3.0          5.8         2.2  virginica
111          6.5         3.2          5.1         2.0  virginica
117          6.5         3.0          5.5         1.8  virginica
148          6.5         3.0          5.2         2.0  virginica
52           6.4         3.2          4.5         1.5 versicolor
75           6.4         2.9          4.3         1.3 versicolor
112          6.4         2.7          5.3         1.9  virginica
116          6.4         3.2          5.3         2.3  virginica
129          6.4         2.8          5.6         2.1  virginica
133          6.4         2.8          5.6         2.2  virginica
138          6.4         3.1          5.5         1.8  virginica
57           6.3         3.3          4.7         1.6 versicolor
73           6.3         2.5          4.9         1.5 versicolor
88           6.3         2.3          4.4         1.3 versicolor
101          6.3         3.3          6.0         2.5  virginica
104          6.3         2.9          5.6         1.8  virginica
124          6.3         2.7          4.9         1.8  virginica
134          6.3         2.8          5.1         1.5  virginica
137          6.3         3.4          5.6         2.4  virginica
147          6.3         2.5          5.0         1.9  virginica
69           6.2         2.2          4.5         1.5 versicolor
98           6.2         2.9          4.3         1.3 versicolor
127          6.2         2.8          4.8         1.8  virginica
149          6.2         3.4          5.4         2.3  virginica
64           6.1         2.9          4.7         1.4 versicolor
72           6.1         2.8          4.0         1.3 versicolor
74           6.1         2.8          4.7         1.2 versicolor
92           6.1         3.0          4.6         1.4 versicolor
128          6.1         3.0          4.9         1.8  virginica
135          6.1         2.6          5.6         1.4  virginica
63           6.0         2.2          4.0         1.0 versicolor
79           6.0         2.9          4.5         1.5 versicolor
84           6.0         2.7          5.1         1.6 versicolor
86           6.0         3.4          4.5         1.6 versicolor
120          6.0         2.2          5.0         1.5  virginica
139          6.0         3.0          4.8         1.8  virginica
62           5.9         3.0          4.2         1.5 versicolor
71           5.9         3.2          4.8         1.8 versicolor
150          5.9         3.0          5.1         1.8  virginica
15           5.8         4.0          1.2         0.2     setosa
68           5.8         2.7          4.1         1.0 versicolor
83           5.8         2.7          3.9         1.2 versicolor
93           5.8         2.6          4.0         1.2 versicolor
102          5.8         2.7          5.1         1.9  virginica
115          5.8         2.8          5.1         2.4  virginica
143          5.8         2.7          5.1         1.9  virginica
16           5.7         4.4          1.5         0.4     setosa
19           5.7         3.8          1.7         0.3     setosa
56           5.7         2.8          4.5         1.3 versicolor
80           5.7         2.6          3.5         1.0 versicolor
96           5.7         3.0          4.2         1.2 versicolor
97           5.7         2.9          4.2         1.3 versicolor
100          5.7         2.8          4.1         1.3 versicolor
114          5.7         2.5          5.0         2.0  virginica
65           5.6         2.9          3.6         1.3 versicolor
67           5.6         3.0          4.5         1.5 versicolor
70           5.6         2.5          3.9         1.1 versicolor
89           5.6         3.0          4.1         1.3 versicolor
95           5.6         2.7          4.2         1.3 versicolor
122          5.6         2.8          4.9         2.0  virginica
34           5.5         4.2          1.4         0.2     setosa
37           5.5         3.5          1.3         0.2     setosa
54           5.5         2.3          4.0         1.3 versicolor
81           5.5         2.4          3.8         1.1 versicolor
82           5.5         2.4          3.7         1.0 versicolor
90           5.5         2.5          4.0         1.3 versicolor
91           5.5         2.6          4.4         1.2 versicolor
6            5.4         3.9          1.7         0.4     setosa
11           5.4         3.7          1.5         0.2     setosa
17           5.4         3.9          1.3         0.4     setosa
21           5.4         3.4          1.7         0.2     setosa
32           5.4         3.4          1.5         0.4     setosa
85           5.4         3.0          4.5         1.5 versicolor
49           5.3         3.7          1.5         0.2     setosa
28           5.2         3.5          1.5         0.2     setosa
29           5.2         3.4          1.4         0.2     setosa
33           5.2         4.1          1.5         0.1     setosa
60           5.2         2.7          3.9         1.4 versicolor
1            5.1         3.5          1.4         0.2     setosa
18           5.1         3.5          1.4         0.3     setosa
20           5.1         3.8          1.5         0.3     setosa
22           5.1         3.7          1.5         0.4     setosa
24           5.1         3.3          1.7         0.5     setosa
40           5.1         3.4          1.5         0.2     setosa
45           5.1         3.8          1.9         0.4     setosa
47           5.1         3.8          1.6         0.2     setosa
99           5.1         2.5          3.0         1.1 versicolor
5            5.0         3.6          1.4         0.2     setosa
8            5.0         3.4          1.5         0.2     setosa
26           5.0         3.0          1.6         0.2     setosa
27           5.0         3.4          1.6         0.4     setosa
36           5.0         3.2          1.2         0.2     setosa
41           5.0         3.5          1.3         0.3     setosa
44           5.0         3.5          1.6         0.6     setosa
50           5.0         3.3          1.4         0.2     setosa
61           5.0         2.0          3.5         1.0 versicolor
94           5.0         2.3          3.3         1.0 versicolor
2            4.9         3.0          1.4         0.2     setosa
10           4.9         3.1          1.5         0.1     setosa
35           4.9         3.1          1.5         0.2     setosa
38           4.9         3.6          1.4         0.1     setosa
58           4.9         2.4          3.3         1.0 versicolor
107          4.9         2.5          4.5         1.7  virginica
12           4.8         3.4          1.6         0.2     setosa
13           4.8         3.0          1.4         0.1     setosa
25           4.8         3.4          1.9         0.2     setosa
31           4.8         3.1          1.6         0.2     setosa
46           4.8         3.0          1.4         0.3     setosa
3            4.7         3.2          1.3         0.2     setosa
30           4.7         3.2          1.6         0.2     setosa
4            4.6         3.1          1.5         0.2     setosa
7            4.6         3.4          1.4         0.3     setosa
23           4.6         3.6          1.0         0.2     setosa
48           4.6         3.2          1.4         0.2     setosa
42           4.5         2.3          1.3         0.3     setosa
9            4.4         2.9          1.4         0.2     setosa
39           4.4         3.0          1.3         0.2     setosa
43           4.4         3.2          1.3         0.2     setosa
14           4.3         3.0          1.1         0.1     setosa

基本統計功能

可以做單一變數統計

x = c(1,2,3,4,5,6,7,8,9,10)
mean(x) 
[1] 5.5
min(x) 
[1] 1
median(x) 
[1] 5.5

基本統計功能 (二)

可以做單一變數統計

max(x) 
[1] 10
var(x)
[1] 9.167
summary(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00    3.25    5.50    5.50    7.75   10.00 

使用sapply

透過sapply針對多個欄位操作

sapply(iris[1:4], mean, na.rm=TRUE)
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
       5.843        3.057        3.758        1.199 

plyr

可用來分割、套用及合併資料的函式

library(plyr)
ddply(iris, c("Species"), function(df) mean(df$Sepal.Length))
     Species    V1
1     setosa 5.006
2 versicolor 5.936
3  virginica 6.588

資料繪圖功能 (一)

Pie Chart

table.iris = table(iris$Species)
pie(table.iris)

plot of chunk unnamed-chunk-14

資料繪圖功能 (二)

Histogram

hist(iris$Sepal.Length)

plot of chunk unnamed-chunk-15

資料繪圖功能 (三)

Box Plot

boxplot(Petal.Width ~ Species, data = iris)

plot of chunk unnamed-chunk-16

資料繪圖功能 (四)

Scatter Plot

plot(x=iris$Petal.Length, y=iris$Petal.Width, col=iris$Species)

plot of chunk unnamed-chunk-17

ggplot2

Scatter Plot

suppressPackageStartupMessages(library(ggplot2))
library(ggplot2)
qplot(Sepal.Length, Petal.Length, data = iris, color = Species)

plot of chunk unnamed-chunk-18

googleViz

suppressPackageStartupMessages(library(googleVis))

library(googleVis)
GeoMarker <- gvisGeoChart(Andrew, "LatLong", sizevar='Speed_kt',colorvar="Pressure_mb", options=list(region="US"))

plot(GeoMarker)

alt text

機器學習與資料分析

資料分析

  • 資料處理
  • 資料分析
  • 解釋分析結果

資料分析與應用

  • 統計分析
  • 迴歸分析
  • 資料分群
  • 資料分類
  • 推薦系統
  • 文字探勘

plot of chunk unnamed-chunk-20

影像辨識

alt text

機器學習

Black-box, algorithmic approach to producing predictions or classifications from data

A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E

– Tom Mitchell (1998)

機器學習主題

  • 監督式學習 (Supervised Learning)
    • 迴歸分析 (Regression)
    • 分類問題 (Classification)
  • 非監督式學習 (Unsupervised Learning)
    • 降低維度 (Dimension Reduction)
    • 分群問題 (Clustering)

迴歸分析

Predict one set of numbers given another set of numbers

Given number of friends x, predict how many goods I will receive on each Facebook posts

Scatter Plot

dataset <- read.csv('fbgood.txt',head=TRUE, sep='\t', row.names=1) 
x = dataset$friends 
y = dataset$getgoods 
plot(x,y) 

plot of chunk unnamed-chunk-21

Linear Fit

plot(x,y) 
fit <- lm(y ~ x)
abline(fit, col = 'red', lwd=3)

plot of chunk unnamed-chunk-22

2nd order polinomial fit

plot(x,y) 
polyfit2 <- lm(y ~ poly(x, 2)) 
lines(sort(x), polyfit2$fit[order(x)], col = 2, lwd = 3)

plot of chunk unnamed-chunk-23

3rd order polinomial fit

plot(x,y) 
polyfit3 <- lm(y ~ poly(x, 3)); 
lines(sort(x), polyfit3$fit[order(x)], col = 2, lwd = 3)

plot of chunk unnamed-chunk-24

其他迴歸分析

  • MASS rlm - Robust Regression
  • GLM - Generalized linear Models
  • GAM - Generalized Additive Models

分類問題

Identifying to which of a set of categories a new observation belongs, on the basis of a training set of data

Given features of bank costumer, predict whether the client will subscribe a term deposit

資料敘述

  • Features:

    • age,job,marital,education,default,balance,housing,loan,contact
  • Labels:

    • Customers subscribe a term deposit (Yes/No)

使用LibSVM

library(e1071) 
dataset <- read.csv('bank.csv',head=TRUE, sep=';') 

set.seed(2)
ind <- sample(2, nrow(dataset), replace = TRUE, prob=c(0.7, 0.3))
train = dataset[ind == 1,]
test = dataset[ind == 2,]

model <- svm(y~., data = train, probability = TRUE) 
pred <- predict(model, test[,1:(dim(test)[[2]]-1)], probability = TRUE)

驗證預測能力

table(pred,test[,dim(test)[2]])

pred    no  yes
  no  1196   98
  yes   25   46

Confusion Matrix

library(caret)
confusionMatrix(table(pred,test[,dim(test)[2]]))
Confusion Matrix and Statistics


pred    no  yes
  no  1196   98
  yes   25   46

               Accuracy : 0.91          
                 95% CI : (0.893, 0.925)
    No Information Rate : 0.895         
    P-Value [Acc > NIR] : 0.0333        

                  Kappa : 0.385         
 Mcnemar's Test P-Value : 8.47e-11      

            Sensitivity : 0.980         
            Specificity : 0.319         
         Pos Pred Value : 0.924         
         Neg Pred Value : 0.648         
             Prevalence : 0.895         
         Detection Rate : 0.876         
   Detection Prevalence : 0.948         
      Balanced Accuracy : 0.649         

       'Positive' Class : no            

使用ROC 評估模型

library(ROCR) 
pred.prob <- attr(pred, "probabilities") 
pred.to.roc <- pred.prob[, 2] 
pred.rocr <- prediction(pred.to.roc, as.factor(test[,(dim(test)[[2]])])) 
perf.rocr <- performance(pred.rocr, measure = "auc", x.measure = "cutoff") 
perf.tpr.rocr <- performance(pred.rocr, "tpr","fpr") 
plot(perf.tpr.rocr, colorize=T,main=paste("AUC:",(perf.rocr@y.values)))

plot of chunk unnamed-chunk-28

論文成就 GET

alt text

Support Vector Machine

  • e1071 - LIBSVM
  • kernlab - SVM, RVM and other kernel learning algorithms
  • klaR - SVMlight
  • rdetools - Model selection and prediction

降低維度

Seeks linear combinations of the columns of X with maximal variance

Calculate a new index to measure economy index of each Taiwan city/county

台灣經濟指標

  • 縣市
  • 營利事業銷售額
  • 經濟發展支出佔歲出比例
  • 得收入者平均每人可支配所得

–2012年《天下雜誌》幸福城市大調查 - 第505期

主成分柱狀圖

dataset <- read.csv('eco_index.csv',head=TRUE, sep=',', row.names=1) 
pc.cr <- princomp(dataset, cor = TRUE) 
plot(pc.cr) 

plot of chunk unnamed-chunk-29

主成分折線圖

screeplot(pc.cr, type="lines") 
abline(h=1, lty=3) 

plot of chunk unnamed-chunk-30

PCA BiPlot

biplot(pc.cr)

plot of chunk unnamed-chunk-31

PCA BarPlot

barplot(sort(-pc.cr$scores[,1], TRUE))

plot of chunk unnamed-chunk-32

其他降低維度套件

  • kpca - Kernel PCA
  • cmdscale - Multi Dimension Scaling
  • SVD - Singular Value Decomposition
  • fastICA - Independent Component Analysis

分群

物以類聚

Segment customers based on existing features

顧客分群

Clustering by 4 features

  • Visit Time
  • Average Expense
  • Loyalty Days
  • Age

決定群數量

mydata <- read.csv('costumer_segment.txt',head=TRUE, sep='\t') 
mydata <- scale(mydata) 
d <- dist(mydata, method = "euclidean") 
fit <- hclust(d, method="ward") 
plot(fit) 

plot of chunk unnamed-chunk-33

Cutting Trees

plot(fit) 
k1 = 4 
groups <- cutree(fit, k=k1) 
rect.hclust(fit, k=k1, border="red") 

plot of chunk unnamed-chunk-34

KMEANS 分群

fit <- kmeans(mydata, k1) 
plot(mydata, col = fit$cluster) 

plot of chunk unnamed-chunk-35

主成分分析圖

library(cluster) 
clusplot(mydata, fit$cluster, color=TRUE, shade=TRUE, lines=0) 

plot of chunk unnamed-chunk-36

其他分群套件

  • kernlab - Spectral Clustering
  • specc - Spectral Clustering
  • fpc - DBSCAN

分析資料一定要Coding 嗎?

alt text

Rattle

alt text

Big Data

Big Data & R

  • RHADOOP
    • 延展 R的運算能力
    • Hadoop 讓 R 可以進行平行運算
    • 不用重新學習新語言(Java 有一定的學習門檻)
  • RHIPE
    • 並非使用R標準函式
    • 以key-value 形式儲存資料,而非Data Frame
  • RMPI
    • MPI 並行運算的速度及效能較RHadoop 優良
    • 可容錯性低,單台伺服器運行錯誤會導致全部工作失敗

RHadoop

alt text

RHadoop 架構

alt text

Streaming v.s. 原生 Java. 

  • 可以讓開發者以其他語言撰寫Mapper/Reducer(R, python, perl)

  • Mapper, reducer, 及 combiner 程序可以用來導向I/O

  • Streaming 會因為要開啟腳本語言的虛擬機器,因此會有額外的時間與空間負擔

Thank You

Author: David Chiu

Contact: david@largitdata.com